Methods and apparatus for documenting a procedure

ABSTRACT

Methods and apparatus are disclosed for documenting a procedure in a touchless environment. Example methods disclosed herein include identifying a position of one or more of a user&#39;s fingertips using a camera of a mobile device and an entropy-based segmentation of an image input, determining a correspondence between user fingertip position and a control interface of the mobile device, including a plurality of functions to be executed based on user input, triggering the mobile device to execute one or more of the plurality of functions based on the user fingertip position during a procedure, and documenting one or more events of the procedure based on the user input for generating a report to be stored in a storage device of the mobile device.

FIELD OF THE DISCLOSURE

This disclosure relates generally to interacting with a mobile device,and, more particularly, to documenting procedures using a mobile device.

BACKGROUND

Mobile devices, such as tablet computers, personal computers (PC),smartphones, mp3 players, etc., have developed many uses. These devicestypically include a touchscreen and/or keyboard for user control of thedevice as well as a camera for taking pictures and/or videos.

In some environments, using a touchscreen or keyboard to control thedevice is not practical. For example, an operating room of a healthcarefacility must remain a sterile environment, and therefore, the use of atouchscreen and/or keyboard is very cumbersome and limited to the riskof infection, contamination by touching the screen. Accordingly, asterile environment does not allow for several users (e.g. nurses,surgeons, anesthesiologist, etc.) to operate such a device in thatparticular environment.

BRIEF SUMMARY

An example method disclosed herein includes identifying a position of auser's fingertips in a virtual plane formed apart from a controlinterface of a mobile device using a camera of the mobile device and anentropy-based segmentation of an image input from the camera,determining a correspondence between user fingertip position and thecontrol interface of the mobile device, the correspondence forming auser input, the control interface including a plurality of functions tobe executed based on the user input, facilitating execution, via themobile device, of one or more of the plurality of functions based on theuser fingertip position in the virtual plane during a procedure, anddocumenting one or more events of the procedure based on the user inputto generate a report to be output via the mobile device.

An example apparatus disclosed herein includes an image filter toidentify a position of one or more of fingertips in a virtual planeformed apart from a control interface of a mobile device using a cameraof the mobile device and an entropy-based segmentation of an image inputfrom the camera, a user command identifier to determine a correspondencebetween user fingertip position and the control interface of the mobiledevice, the correspondence forming a user input, the control interfaceincluding a plurality of functions to be executed based on the userinput, a controller to facilitate execution, via the mobile device, ofone or more of the plurality of functions based on the user fingertipposition in the virtual plane during a procedure, and a documenter todocument one or more events of the procedure based on the user input togenerate a report to be output via the mobile device.

An example machine medium comprising instructions which when executedcause a machine to identify a position of one or more of a user'sfingertips in a virtual plane formed apart from a control interface of amobile device using a camera of the mobile device and an entropy-basedsegmentation of an image input from the camera, determine acorrespondence between user fingertip position and the control interfaceof the mobile device, the correspondence forming a user input, thecontrol interface including a plurality of functions to be executedbased on the user input, facilitate execution, via the mobile device, ofone or more of the plurality of functions based on the user fingertipposition in the virtual plane during a procedure, and document one ormore events of the procedure based on the user input to generate areport to be output via the mobile device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example environment of use for a touchless methodof interacting with a mobile device in accordance with the teachings ofthis disclosure.

FIG. 2 is a block diagram of an example mobile device constructed inaccordance with the teachings of this disclosure.

FIG. 3 is a block diagram of an example controller and an exampledocumenter for use in the example mobile device of FIG. 2.

FIG. 4 is a flowchart representative of first example machine readableinstructions that may be executed to implement the example controllerand/or documenter of FIGS. 2 and/or 3.

FIG. 5 is a flowchart representative of second example machine readableinstructions that may be executed to implement the example documenter ofFIGS. 2 and/or 3.

FIG. 6 is a flowchart representative of third example machine readableinstructions that may be executed to implement an example deviceidentifier of the example documenter of FIG. 3.

FIGS. 7A-7C illustrate example finger positions and/or gestures forcontrol of the example mobile device of FIGS. 1 and/or 2.

FIG. 8 is a block diagram of an example processor platform that mayexecute the instructions of FIGS. 4, 5, and/or 6 to implement theexample controller of FIGS. 2 and/or 3.

DETAILED DESCRIPTION

Certain examples provide real time (or substantially real timeaccounting for processing and/or storage delay) input from an ongoingprocedure for documentation of procedure events occurring in anenvironment in which control of a mobile device using a touchscreenand/or keyboard is not practical. Accordingly, allowing users in anenvironment to control a single device to avoid repeating documentationand allow for increased accuracy and efficiency is provided in thisdisclosure.

Additionally, in many environments, in particular an operating room of ahealthcare facility, it may be difficult or impossible for a user tocontrol a mobile device to document a procedure (e.g., tablet computer,smartphone, personal digital assistant, laptop computer etc.) using thestandard control interface (touchscreen, keyboard, mouse, etc.) of themobile device. Various reasons for the inability to control the mobiledevice using the standard control interface may include keeping theuser's hands sterile and/or keeping device clean from debris (e.g.,blood, dirt, oil, etc.). Accordingly, having the ability to control themobile device to document the procedure without touching the mobiledevice would be desirable.

Example methods and apparatus disclosed allow a user to control themobile device to document a procedure without touching the mobile deviceusing the camera of the mobile device and an entropy-based segmentationof images captured by the camera. The camera captures images of theuser's hands, and the entropy-based segmentation applied to the imagesis used to identify the user's fingertip location. The use ofentropy-based segmentation allows for any hand, whether it is from thesame user or several users, to be identified even if it is covered indebris or soiled. A virtual plane may be established for identificationof the one or more user's fingertips. The virtual plane can be used asan “in air” control interface corresponding to the control interface(e.g., touchscreen, display, etc.) of the mobile device. The user maythen control the mobile device using the virtual plane as the controlinterface, and accordingly, the user can then access some or allfunctions (e.g., browsers, applications, etc.) of the mobile device.

Additionally, it may be desirable to document the instruments or devicesthat a user is using for a certain procedure, without having to indicatethe instrument or device using standard control interface. Examplemethods and apparatus disclosed identify the instruments or devicesusing the camera of the mobile device. The images received from thecamera are monitored by a device detector that detects or may recognizelabels, bar codes, and/or shapes of the instruments or devices beingused in the procedure. The user may present the device to the camera foridentification, and the mobile device documents the use of the deviceand when it is being used during the procedure.

Example methods and apparatus disclosed enable a mobile device toreceive documentation data as described above and compile the data togenerate a report of the procedure. The documentation data may becompiled after being converted to an appropriate format. In someexamples, the documentation data includes transcribed voice to text datato report any instructions, discussions, and/or user commands madeduring the procedure. Additionally, the mobile device may record audio,video, and/or images of the procedure and provide the audio capture,video capture, and/or image capture with the generated report.

Thus, the disclosed methods and apparatus allow a user to interact witha mobile device to document a procedure with substantially no need touse the standard control interface of the mobile device.

FIG. 1 illustrates an example environment 100, such as an operating roomof a healthcare facility, in which an example mobile device 110 may beused. In the example environment 100, an operating table 102, users 104,106, a mobile device 110, and a virtual plane 120 are shown. In theenvironment 100, the users 104, 106 may perform a procedure on operatingtable 102. In the illustrated example, the mobile device 110 is locatedwithin a viewing distance from the users 104, 106. In some examples, themobile device can be one or more of a tablet computer, cellular phone,smartphone, personal digital assistant (PDA), personal computer (PC),laptop computer, etc. Furthermore, in the illustrated example, theenvironment 100 is a sterile environment, in which the users 104, 106should not touch the mobile device 110 so as not to spread anycontaminates on the users 104, 106 and/or the mobile device 100. In theillustrated example, the mobile device 110 has a touchscreen 112 and acamera 114.

Users 104, 106 can document the procedure being performed in theillustrated example of FIG. 1 using the mobile device 110 withouttouching the mobile device 110 during the procedure. In the illustratedexample, the camera 114 of the mobile device 110 may be used to detectfingertips of the users 104, 106 using entropy-based segmentation. Insome examples, the entropy-based segmentation allows for detection ofthe users' 104, 106 fingertips even if the users 104, 106 are usinggloves and/or if the users' gloves and/or fingertips are covered indebris (e.g. blood, dirt, oils, etc.) or soiled. A virtual plane 120corresponding to the touchscreen 112 of the mobile device 100 may beestablished for the users 104, 106 to control the mobile device 110. Theusers 104, 106 may select icon(s) displayed on the touchscreen 112designating function(s) to be executed on the mobile device 110 bypointing to a location of the virtual plane 120 corresponding to thelocation of the icon(s) on the touchscreen 112.

In some examples, the users 104, 106 of FIG. 1 may use devices during aprocedure that can be identified by the mobile device 100. In someexamples, the mobile device 100 includes a device identifier that usesthe camera 114 of the mobile device 100. The device identifier mayautomatically recognize a device based on one or more types of deviceidentification (e.g. a bar code, label, device shape, etc.). In someexamples, the mobile device 100 decodes the device identification todetermine device information (e.g. device name, type, model, etc.). Insome examples, the mobile device 100 cross-references the deviceidentification with a database stored on the mobile device 100 or atanother location accessible by a network connected to the mobile device100.

In some examples, the mobile device 110 records the procedure performedin the example environment 100 of FIG. 1. A video recording of aprocedure may be performed and associated with a report documenting theprocedure. The mobile device 110 may enable voice identificationsoftware to determine which of the users 104, 106 was speaking at agiven time. Furthermore, in some examples, the mobile device 110 mayemploy a voice to text function to transcribe spoken words of the users104, 106 for generation of a report of the procedure performed in theexample environment 100.

Accordingly, in the example environment 100 of FIG. 1 the users 104, 106may document a procedure using the example mobile device 110 withouthaving to touch the mobile device. For example, the mobile device 100may document the procedure by generating a report that includes controlcommands of mobile device 100 users, identification(s) of device(s) usedduring the procedure, audio capture, video capture, still image captureof the procedure, voice to text transcription(s) ofdiscussion/instruction of user(s) during the procedure, applicationfunction(s) used on the mobile device 100 during the procedure,reference material(s) and/or file(s) accessed by the user(s) during theprocedure, etc. Documentation may combine one or more of these contentitems to form a report, for example.

FIG. 2 is a block diagram of an example mobile device 200 that may beused to implement the mobile device 110 of FIG. 1. The example mobiledevice 200 includes a camera 202, a microphone 204, a control interface206, an application interface 208, a documenter 210, and a controller220. The example mobile device 200 includes a communication bus 201 thatfacilitates communication between the camera 202, the microphone 204,the control interface 206, the application interface 208, the documenter210, and/or the controller 220. In the illustrated example, each of theboxes (202-220) may be combined, rearranged etc. For example, the camera202 may be combined with the microphone 204. In some examples, thecamera 202 receives example images and the microphone 204 receivesexample audio. In some examples, the control interface 206 receivesinput from a user (e.g. a touchscreen, keypad, etc.) and sends output tothe user (e.g. a display, etc.). The application interface 208 may allowuser(s), via the control interface 206, and or the controller 220 tointeract with and/or execute applications and/or functions that may bestored on the mobile device 200. In some examples, the exampledocumenter 210 may store and/or record user interactions with the mobiledevice 200. The documenter 210 may generate a report of events based onthe user interactions for a procedure being performed by the user.

The example controller 220 may be used to execute the commands of a user(e.g. the users 104, 106) received via the user interface 206. Thecontroller 220 may also receive commands from the users (e.g., the users104, 106) based on a fingertip positions identified by the camera 202using an example entropy-based segmentation algorithm. In some examples,the controller 220 receives the images from the camera 202 and appliesthe entropy-based segmentation algorithm to the images. Having appliedthe example algorithm, the controller 220 identifies the users 104, 106fingertip locations in the images. After applying the example algorithmto a series of images received from the camera 202, the examplecontroller can interpret commands from the users 104, 106 based on thefingertip locations. Some commands may include, but are not limited to,controlling functions and/or applications of the mobile device 200,requesting identification of a device, operating a browser of the mobiledevice 200, operating the camera and/or microphone of the mobile device200, etc.

FIG. 3 is a block diagram of an example controller 220 and an exampledocumenter 210 which may be used to implement the controller 220 anddocumenter 210 of FIG. 2, respectively. In the illustrated example, theexample controller 220 of FIG. 3 includes a device detector 322, adevice identifier 324, an image filter 326, a user command identifier328, and a voice recognizer 330. In the illustrated example, the exampledocumenter 210 includes a documentation compiler 312 and a reportgenerator 314.

In FIG. 3, the example controller 220 receives example images from acamera (e.g. the camera 202), audio from a microphone (e.g. themicrophone 204), and control interface data from a control interface(e.g. the control interface 206). In the illustrated example, images arereceived by the device detector 322 and the image filter 326.

The example device detector 322 of FIG. 3 may recognize a uniqueidentifier of a device that is being used during an example procedure.For example, in a medical procedure, devices may include implantsinstalled and/or tools used during the medical procedure. The devicedetector 322 may automatically recognize a label, barcode, and/or shapeof the example devices. Once the device detector 322 recognizes an imageof a device identification, the device detector 322 forwards the imageto the device identifier 324 to determine device information based onthe image. The device identifier 324 may decode the image to obtain thedevice information. For example, the device identification may include abar code. The device identifier 324 may use a scanning software/functionto determine the device information. In some examples, the deviceidentifier 324 may decode the image to identify text in the image, suchas labels and/or model numbers on the device. In some examples, thecontroller 220 and/or device identifier 324 may then cross reference theimage with device information stored in a database of an example mobiledevice (e.g. the mobile device 200) to identify device information. Oncethe device identifier 324 determines the device information, the deviceinformation forwards the information to documenter 210 (perhaps via acommunication bus, such as communication bus 201).

The example image filter 326 of FIG. 3 receives images from the camera(e.g. the camera 202). The image filter 326 focuses on texture basedregions of an image for segmentation of the hands, fingers, and/orfingertips with the background. After applying an entropy-basedsegmentation algorithm to the images, a min-max filter may be applied toidentify the fingers. Once the hands, fingers, and/or fingertips of anexample user (e.g. the users 104, 106) are identified using the imagefilter 326 for the images, a user command identifier 328 of FIG. 3determines a user command from the filtered images.

The user command identifier 328 of FIG. 3 tracks the user's hands,fingers, and/or fingertips in the images based on the segmentation ofthe images by the image filter 326. By determining the location of theuser's hands, fingers, and/or fingertips in each of the images, the usercommand identifier 328 can interpret the commands of the user bycross-referencing command data stored in a storage device of the examplemobile device 200 and/or a database connected to a network in connectionwith the mobile device. In some examples, the controller 220 uses thecommands identified by the user command identifier to executeapplications and/or functions of the mobile device 200. The user commandidentifier 328 may forward the identified commands to the documenter 210for documentation.

The voice recognizer 330 of FIG. 3 receives audio from a microphone ofthe example mobile device 200. The voice recognizer 330 may filter thevoice data and forward the filtered data to the user command identifier328 to determine a command from the user. In some examples, thecontroller 220 may use the voice command data from the voice recognizer330 to control the example mobile device. In some examples, the voicerecognizer 330 converts the voice data to text and forwards the voicedata to the documenter 210 for documentation. In certain examples, thevoice recognizer 330 has voice recognition software, and may identifyone or more users (e.g., the users 104, 106) based on the voice receivedover the audio. The voice recognizer 330 may associate the voice data toeach of the example users and forward the information to the documenter210. In some examples, the documenter 210 documents the voice data andassociates the voice data with each corresponding user to document whichuser voiced which command and/or information about an example procedure.

The documenter 210 of FIG. 3 receives data from the controller 220 viathe device identifier 324, the user command identifier 328, and/or thevoice recognizer 330. In some examples, the documenter 210 receivescontrol interface data from the control interface 206. The documenter210 receives information at the documentation compiler 312. Thedocumentation compiler 312 compiles the data to be documented by theexample mobile device 200. In some examples, the documentation compiler312 sorts the information based on the time the information was receivedat the documentation compiler 312. In some examples, the documentationcompiler 312 converts voice and/or image data received from the voicerecognizer 330, user command identifier 328, and/or device identifier324 to text data. In some examples, the documentation compiler thencompiles the text data based on a time associated with receiving thedata from the user. In some examples, the documentation compiler 312forwards the compiled data to the report generator 314 for generation ofa report to a user.

The example report generator 314 of FIG. 3 generates reports forpresentation to a user. In some examples, the user may customize thereport via a control interface (e.g. the control interface 206), basedon time, location, user, procedure, information, etc. In some examples,the report generator sorts data from the documentation compiler 312based on data type (e.g., audio data, device data, command data (by handand/or voice), etc.). In some examples the report generator may retrievea video recorded by a camera (e.g. the camera 202) of a procedure, andassociate the video with the generated report. In some examples, thegenerated report is displayed via a display (e.g. a touchscreen) of amobile device (e.g. the mobile device 200). In some examples, theexample report is stored in an example database of the example mobiledevice and/or a database connected to a network in connection with themobile device.

While an example manner of implementing the example mobile device 200 ofFIG. 2 and more specifically, the example controller 220 and exampledocumenter 210 of FIG. 3, have been illustrated in FIGS. 2 and/or 3, oneor more of the elements, processes and/or devices illustrated in FIGS. 2and/or 3 may be combined, divided, re-arranged, omitted, eliminatedand/or implemented in any other way. Further, the example devicedetector 322, the example device identifier 324, the example imagefilter 326, the example user command identifier 328, the example voicerecognizer 330, the example documentation compiler 312, and the examplereport generator 314 and/or, more generally, the example controller 220and the example documenter 210 of FIGS. 2 and/or 3 may be implemented byhardware, software, firmware and/or any combination of hardware,software and/or firmware. Thus, the example device detector 322, theexample device identifier 324, the example image filter 326, the exampleuser command identifier 328, the example voice recognizer 330, theexample documentation compiler 312, and the example report generator 314and/or, more generally, the example controller 220 and the exampledocumenter 210 could be implemented by one or more circuit(s),programmable processor(s), application specific integrated circuit(s)(ASIC(s)), programmable logic device(s) (PLD(s)) and/or fieldprogrammable logic device(s) (FPLD(s)), etc. When any of the apparatusclaims of this patent are read to cover a purely software and/orfirmware implementation, at least one of, the example device detector322, the example device identifier 324, the example image filter 326,the example user command identifier 328, the example voice recognizer330, the example documentation compiler 312, and the example reportgenerator 314 are hereby expressly defined to include a computerreadable storage medium such as a memory, DVD, CD, etc. storing thesoftware and/or firmware. Further still, the example controller 220 anddocumenter 210 may include one or more elements, processes and/ordevices in addition to, or instead of, those illustrated in FIGS. 2and/or 3, and/or may include more than one of any or all of theillustrated elements, processes and devices.

Flowcharts representative of example machine readable instructions forimplementing the controller 220 and the documenter 230 of FIGS. 2 and/or3 are shown in FIGS. 4-6. In this example, the machine readableinstructions comprise program(s) for execution by a processor such asthe processor 812 shown in the example processor platform 800 discussedbelow in connection with FIG. 8. The program may be embodied in softwarestored on a tangible computer readable storage medium such as a CD-ROM,a floppy disk, a hard drive, a digital versatile disk (DVD), a BLU-RAY™disc, or a memory associated with the processor 812, but the entireprogram and/or parts thereof could alternatively be executed by a deviceother than the processor 812 and/or embodied in firmware or hardware.Further, although the example program is described with reference to theflowcharts illustrated in FIGS. 4-6, many other methods of implementingthe example controller 220 and documenter 210 of FIGS. 2 and/or 3 mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined.

As mentioned above, the example processes of FIGS. 4-6 may beimplemented using coded instructions (e.g., computer readableinstructions) stored on a tangible computer readable storage medium suchas a hard disk drive, a flash memory, a read-only memory (ROM), acompact disk (CD), a digital versatile disk (DVD), a BLU-RAY™ disc, acache, a random-access memory (RAM)) and/or any other storage medium inwhich information is stored for any duration (e.g., for extended timeperiods, permanently, brief instances, for temporarily buffering, and/orfor caching of the information). As used herein, the term tangiblecomputer readable storage medium is expressly defined to include anytype of computer readable storage device or disc and to excludepropagating signals. Additionally or alternatively, the exampleprocesses of FIGS. 4-6 may be implemented using coded instructions(e.g., computer readable instructions) stored on a non-transitorycomputer readable storage medium such as a hard disk drive, a flashmemory, a read-only memory, a compact disk, a digital versatile disk, acache, a random-access memory and/or any other storage medium in whichinformation is stored for any duration (e.g., for extended time periods,permanently, brief instances, for temporarily buffering, and/or forcaching of the information). As used herein, the term non-transitorycomputer readable storage medium is expressly defined to include anytype of computer readable storage device or disc and to excludepropagating signals.

Example machine readable instructions 400 that may be executed toimplement the controller 220 and/or documenter 210 of FIGS. 2 and/or 3are represented by the flowchart shown in FIG. 4. The machine readableinstructions 400 of FIG. 4, upon execution, cause the controller 220 anddocumenter 210 of an example mobile device 200 to documents events of aprocedure using based on one or more users interacting with the mobiledevice in a touchless environment, such as example environment 100 ofFIG. 1. At block 410, an example mobile device, such as mobile device200 of FIG. 2, is setup for use during a procedure in an exampletouchless environment, such as example environment 100. In someexamples, the mobile device 100 is setup using a user interface prior tobringing the mobile device 200 into the touchless environment 100. Insome examples, the setup of the mobile device 200 includes executing anexample documentation application stored on the mobile device 200.

At block 420, position of a user's fingertip(s) is identified. Forexample, the camera 202 of the example mobile device 200 is used toidentify a user's fingertips. The image filter 326 receives images fromthe camera 202. The image filter 326 applies an entropy-basedsegmentation to the images followed by, for example, min-max filter toidentify one or more user's fingertips.

In some examples, an initial identification of a user's fingertips takesplace following execution of an example documentation application ofblock 410. In some examples, a user establishes an example virtual planecorresponding to a control interface 206 (e.g. a display, touchscreen,etc.) of the mobile device 200. The virtual plane may be established byperforming a calibration, wherein the user identifies an upper-leftcorner and a lower-right corner of the virtual plane and/or anupper-right corner and a lower-left corner of the virtual plane uponexecution of an example documentation application, for example.

At block 430 of the illustrated example, a correspondence between theuser's fingertip position within an established virtual plane isdetermined. For example, the user command identifier 328 of the examplecontroller 220 may determine a correspondence between a user's fingertipposition within an established virtual plane using the camera 202 of themobile device 200. At block 430, a tracking is performed using a pixelshift algorithm for fast tracking of segmented zones of the imagesreceived from the image filter 326. In some examples, horizontal andvertical tracking (see, e.g., FIG. 7C) is used for navigation within anexample application. In block 430, using a texture based entropysegmentation allows for an example user's hands, fingers, and/orfingertips and/or the gloves covering them to become soiled or coveredin debris (e.g. blood, dirt, oils, etc.) during a procedure withoutaffecting the image filter's 326 ability to identify the user's hands,fingers, and/or fingertips and their location. At block 430, the usercommand identifier 328 track's the user's fingertip position within thevirtual plane and corresponds that position to a location on the controlinterface 206 of the mobile device 200.

At block 440 of FIG. 4, the user command identifier 328 determines thatthe user has input a command based on the user's fingertip position, andin some examples, based on the user's fingertip position in an examplevirtual plane. In some examples, an example tracking of the user'sfingertips in images received from the image filter 326 identifies auser's action of touching an up control by estimation of the user'sfingertip distance to the virtual plane and/or the control interface 206of the mobile device 200. In some examples, the estimation is a matchthe matter—object forward and then object backward with a timingvariable. Using match-the-matter estimation, the analyzed patters may bemapped to touch events for execution of an application. An icon forexecuting the application may be displayed on the control interface 206at a location corresponding to the user's fingers location in theexample virtual plane.

At block 450, an example documenter 210 of the mobile device 200documents events of a procedure based upon a user's input andinteraction with the mobile device 200. In some examples, a user mayselect which events of a procedure are to be documented by documenter210. In some examples, the documenter automatically documents allcommands and information received by any interactions of any users ofthe mobile device 200. Instructions of block 450 are further explainedin FIG. 5.

At block 460 of FIG. 4, the example documenter 210 may generate a reportvia report generator 314. In some examples, the report generator sortsthe data from the documentation compiler 312 based on data type (e.g.audio data, device data, command data, etc.). In some examples, thereport generator retrieves a video of a procedure recorded by the camera202 and associates the video with the generated report. In someexamples, the generated report is displayed via a display (e.g. controlinterface 206) of the mobile device 200 and/or the example report may bestored in an example database of the example mobile device and/or adatabase connected to a network in connection with the mobile device.

FIG. 5 illustrates example machine readable instructions 450 that may beexecuted to implement block 450 of FIG. 4. The instructions 450 beginwhen the documenter 210 receives data information for documentation atblock 510. At block 510, the documenter 210 may receive data informationfor documentation from the device identifier 324 (see FIG. 6), the usercommand identifier 328, the voice recognizer 330, and/or the controlinterface 206. Accordingly, the data received may be image data, voicedata, metadata, text data, or any form of data.

At block 520, the document compiler 312 determines whether the dataneeds to be converted before compiling. The documenter 210 may determinewhether to convert the received data (e.g. from voice to text and/orimage to metadata) based on a user's input from the control interface206 and/or settings set by the user. If the received data does not needto be converted, control moves to block 540, at which the data is storedin a storage device of the mobile device 200 by the documentationcompiler 312. In some examples, a user may indicate that alldocumentation data be converted to text in the documentation compiler312. If the received data does need to be converted, the documentationcompiler 312 then converts the data based on setting(s) corresponding tothe type of data received. Image data from the device identifier may beconverted to text data identifying the device information, and/or voicedata received from the voice recognizer may be transcribed to text data,for example. At block 540, all received documentation data should beconverted to the correct format prior to being stored in an examplestorage device of the mobile device 200 or an example storage deviceconnected to a network in communication with the mobile device 200.

In some examples, the documentation compiler 312 of documenter 210 sortsthe documentation data based on at least one of data type or time. Insome examples, the sorted data may be associated with a video recordingcorresponding to the documentation data. In some examples, thedocumentation data is received by the mobile device 200 while a camera202 of the mobile device 200 was recording the video.

FIG. 6 illustrates example machine readable instructions 600 that may beexecuted to implement the device detector 322 and device identifier ofFIG. 3. At block 610 of the illustrated example, the device detector 322receives images from the example camera 202. At block 620, the devicedetector 322 determines whether a device (e.g., implant, tool, etc.) islocated within the image received. The device detector 322 may detect adevice based on a label, bar code, shape or any other distinguishingfeatures of a device identified in the images. If the device detector322 does not detect a device in the image, control returns to block 610to receive the next image from the camera 202.

If the device detector 322 does detect a device at block 620, the deviceidentifier 324 at block 630 may decode and/or extract deviceidentification information (e.g., a bar code, label, etc.) from theimage containing the identified device. At block 640, the deviceidentifier 324 may then determine device information (e.g., the devicename, model number, etc.) based on the identification information. Insome examples, the device identifier 324 cross checks the identificationinformation with stored information on a storage drive of the mobiledevice to determine the identification information. In some examples,the device identifier 324 references a database of a storage deviceconnected to a network (e.g., a local area network or the Internet) todetermine the device information. At block 650, the device identifier324 then forwards the device information to the documenter 210 fordocumentation of the identified device. In some examples, the deviceidentifier 324 forwards an image of the device along with the deviceinformation for documentation.

FIGS. 7A-7C are illustrated examples of fingertip movements and/orpositions that may be used to control the mobile device 200. FIGS. 7A-7Ceach show an example mobile device 700A, 700B, 700C (which may beimplemented by the mobile device 200 of FIG. 2), an example virtualplane 710A, 710B, 710C, and an example user hand position 720A, 720B,720C, respectively. In FIGS. 7A-7C the example virtual planes 710A,710B, 710C are a distance Z from the example mobile devices 700A, 700B,700C. In FIGS. 7A-7C, corresponding dotted lines identify that thecorners of the virtual planes 710A, 710B, 710C correspond to thecorresponding corners of a control interface 706A, 706B, 706C of themobile device 700A, 700B, 700C, respectively.

FIG. 7A illustrates a virtual plane 710A having been established adistance Z from the mobile device 700A. When the user hand 720Ainteracts with the virtual plane 710A, an example camera (e.g. thecamera 202) of the mobile device 700A forwards images to an image filter326 to determine the hand, finger, and/or fingertip position relative tothe virtual plane 710A. From there, an example user command identifiercan interpret commands based on the positions of the user's hand 720A.

FIG. 7B illustrates an example command made by a user to zoom in adisplay of the control interface 706B. In FIG. 7B, the user hand 720Bmay “pinch” the virtual screen to zoom in and/or the control interface706B of the mobile device 700B. FIG. 7C illustrates an example commandmade by a user to scroll up and/or down a page being displayed on thecontrol interface 706C of the user device 700C. FIGS. 7A-7C are strictlyconsidered to be example hand gestures or positions to control themobile devices 700A, 700B, 700C and may be modified to implemented otherfunctions or controls of the mobile devices 700A, 700B, 700C.

FIG. 8 is a block diagram of an example computer 800 capable ofexecuting the instructions of FIGS. 4-6 to implement the controller 220and documenter 210 of FIGS. 2 and/or 3. The computer 800 can be, forexample, a server, a personal computer, a mobile phone (e.g., a cellphone), a smartphone, a table personal computer, a personal digitalassistant (PDA), a digital video recorder, a personal video recorder, aset top box, or any other type of computing device.

The system 800 of the instant example includes a processor 812. Forexample, the processor 812 can be implemented by one or more Intel®microprocessors from the Pentium® family, the Itanium® family or theXScale® family. Of course, other processors from other families are alsoappropriate.

The processor 812 is in communication with a main memory including avolatile memory 814 and a non-volatile memory 816 via a bus 818. Thevolatile memory 814 may be implemented by Synchronous Dynamic RandomAccess Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUSDynamic Random Access Memory (RDRAM) and/or any other type of randomaccess memory device. The non-volatile memory 816 may be implemented byflash memory and/or any other desired type of memory device. Access tothe main memory 814, 816 is typically controlled by a memory controller(not shown).

The computer 800 also includes an interface circuit 820. The interfacecircuit 820 may be implemented by any type of interface standard, suchas an Ethernet interface, a universal serial bus (USB), and/or a PCIexpress interface.

One or more input devices 822 are connected to the interface circuit820. The input device(s) 822 permit a user to enter data and commandsinto the processor 812. The input device(s) can be implemented by, forexample, a keyboard, a mouse, a touchscreen, a track-pad, a trackball,isopoint and/or a voice recognition system.

One or more output devices 824 are also connected to the interfacecircuit 820. The output devices 824 can be implemented, for example, bydisplay devices (e.g., a liquid crystal display). The interface circuit820, thus, typically includes a graphics driver card.

The interface circuit 820 also includes a communication device, such asa modem or network interface card to facilitate exchange of data withexternal computers via a network 826 (e.g., an Ethernet connection, adigital subscriber line (DSL), a telephone line, coaxial cable, acellular telephone system, etc.).

The computer 800 also includes one or more mass storage devices 828 forstoring software and data. Examples of such mass storage devices 828include floppy disk drives, hard drive disks, compact disk drives anddigital versatile disk (DVD) drives.

The coded instructions of FIGS. 4-6 may be stored in the mass storagedevice 828, in the volatile memory 814, in the non-volatile memory 816,and/or on a removable storage medium such as a CD or DVD

From the foregoing, it will appreciate that the above disclosed methods,apparatus and articles of manufacture allow for touchless control of amobile device in a touchless environment and documentation of aprocedure by that mobile device based on input control using a camera ofthe mobile device, audio recording, and/or video recording, as well asdevice implant and/or tool recognition abilities.

Although certain example methods, apparatus and articles of manufacturehave been described herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

What is claimed is:
 1. A documenting procedure method comprising:identifying a position of one or more of a user's fingertips in avirtual plane formed apart from a control interface of a mobile deviceusing a camera of the mobile device and an entropy-based segmentation ofan image input from the camera; determining a correspondence betweenuser fingertip position and the control interface of the mobile device,the correspondence forming a user input, the control interface includinga plurality of functions to be executed based on the user input;facilitating execution, via the mobile device, of one or more of theplurality of functions based on the user fingertip position in thevirtual plane during a procedure; and documenting one or more events ofthe procedure based on the user input to generate a report to be outputvia the mobile device.
 2. A method according to claim 1, wherein thevirtual plane is separate from and corresponds to the control interfaceof the mobile device, and wherein first coordinates of the virtual planecorrespond to second coordinates of the control interface of the mobiledevice.
 3. A method according to claim 1, further comprising detecting adevice identification in the image input using the camera of the mobiledevice; determining device information based on the deviceidentification; and documenting the device information in the report asone of the one or more events of the procedure.
 4. A method according toclaim 1, further comprising storing at least one of an audio recordingor video recording of the procedure using at least one of a microphoneor the camera based on a user command identified by the user fingertipposition; and associating the recording of the procedure with thereport.
 5. A method according to claim 1, wherein the user's fingertipposition is to be identified at a first time when one or more glovescovering the user's fingertips are substantially clear of debris and ata second time when the one or more gloves covering the user's fingertipsare substantially covered in debris.
 6. A method according to claim 1,wherein documenting the one or more events further comprises convertingdocumentation data from the execution of the one or more functions to aformat based on at least one of a user setting or data type.
 7. A methodaccording to claim 1, wherein the mobile device is at least one of atablet computer, smartphone, personal digital assistant, personalcomputer, or laptop computer.
 8. An apparatus for documenting procedurecomprising: an image filter to identify a position of one or more offingertips in a virtual plane formed apart from a control interface of amobile device using a camera of the mobile device and an entropy-basedsegmentation of an image input from the camera; a user commandidentifier to determine a correspondence between user fingertip positionand the control interface of the mobile device, the correspondenceforming a user input, the control interface including a plurality offunctions to be executed based on the user input; a controller tofacilitate execution, via the mobile device, of one or more of theplurality of functions based on the user fingertip position in thevirtual plane during a procedure; and a documenter to document one ormore events of the procedure based on the user input to generate areport to be output via the mobile device.
 9. An apparatus according toclaim 8, wherein the virtual plane is separate from and corresponds tothe control interface of the mobile device, and wherein firstcoordinates of the virtual plane correspond to second coordinates of thecontrol interface of the mobile device.
 10. An apparatus according toclaim 8, further comprising a device identification detector to detect adevice identification in the image input using the camera of the mobiledevice; and a device identifier to determine device information based onthe device identification, wherein the documenter documents the deviceinformation in the report as one of the one or more events of theprocedure.
 11. An apparatus according to claim 8, wherein the controllerstores at least one of an audio recording or video recording of theprocedure using at least one of a microphone or the camera based on auser command identified by the user fingertip position and thedocumenter associates the recording of the procedure with the report.12. An apparatus according to claim 8, wherein the user's fingertipposition is identified at a first time when one or more gloves coveringthe user's fingertips are substantially clear of debris and at a secondtime when the one or more gloves covering the user's fingertips aresubstantially covered in debris.
 13. An apparatus according to claim 8,wherein documenting the one or more events further comprises convertingdocumentation data from the execution of the one or more functions to aformat based on at least one of a user setting or data type.
 14. Anapparatus according to claim 8, wherein the mobile device is at leastone of a tablet computer, smartphone, personal digital assistant,personal computer, or laptop computer.
 15. A non-transitory machinereadable storage medium comprises instructions which, when executed,cause a machine to at least: identify a position of one or more of auser's fingertips in a virtual plane formed apart from a controlinterface of a mobile device using a camera of the mobile device and anentropy-based segmentation of an image input from the camera; determinea correspondence between user fingertip position and the controlinterface of the mobile device, the correspondence forming a user input,the control interface including a plurality of functions to be executedbased on the user input; facilitate execution, via the mobile device, ofone or more of the plurality of functions based on the user fingertipposition in the virtual plane during a procedure; and document one ormore events of the procedure based on the user input to generate areport to be output via the mobile device.
 16. A non-transitory machinereadable storage medium according to claim 15, wherein the virtual planeis separate from and corresponds to the control interface of the mobiledevice, and wherein first coordinates of the virtual plane correspond tosecond coordinates of the control interface of the mobile device.
 17. Anon-transitory machine readable storage medium according to claim 15,wherein the instructions cause the machine to detect a deviceidentification in the image input using the camera of the mobile device;determine device information based on the device identification; anddocument the device information in the report as one of the one or moreevents of the procedure.
 18. A non-transitory machine readable storagemedium according to claim 15, wherein the instructions cause the machineto store at least one of an audio recording or video recording of theprocedure using at least one of a microphone or the camera based on auser command identified by the user fingertip position; and associatethe recording of the procedure with the report.
 19. A non-transitorymachine readable storage medium according to claim 15, wherein theuser's fingertip position is identified at a first time when one or moregloves covering the user's fingertips are substantially clear of debrisand at a second time when the one or more gloves covering the user'sfingertips are substantially covered in debris.
 20. A non-transitorymachine readable storage medium according to claim 15, whereindocumenting the one or more events further comprises convertingdocumentation data from the execution of the one or more functions to aformat based on at least one of a user setting or data type.